High-dimensional dynamics of generalization error in neural networks
نویسندگان
چکیده
منابع مشابه
High-dimensional dynamics of generalization error in neural networks
We perform an average case analysis of the generalization dynamics of large neural networks trained using gradient descent. We study the practically-relevant “high-dimensional” regime where the number of free parameters in the network is on the order of or even larger than the number of examples in the dataset. Using random matrix theory and exact solutions in linear models, we derive the gener...
متن کاملTemporal Dynamics of Generalization in Neural Networks
This paper presents a rigorous characterization of how a general nonlinear learning machine generalizes during the training process when it is trained on a random sample using a gradient descent algorithm based on reduction of training error . It is shown, in particular, that best generalization performance occurs, in general, before the global minimum of the training error is achieved. The dif...
متن کاملGeneralization Error of Limear Neural Networks in Unidentifiable Cases
The statistical asymptotic theory is often used in many theoretical results in computational and statistical learning theory. It describes the limiting distribution of the maximum likelihood estimator as an normal distribution. However, in layered models such as neural networks , the regularity condition of the asymptotic theory is not necessarily satissed. If the true function is realized by a...
متن کاملBounding the Generalization Error of Neural Networks and Combined Classifiers
Recently, several authors developed a new approach to bounding the generalization error of complex classi-ers (of large or even innnite VC-dimension) obtained by combining simpler classiiers. The new bounds are in terms of the distributions of the margin of combined classiiers and they provide some theoretical explanation of generalization performance of large neu-We obtained new probabilistic ...
متن کاملOpening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks
Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes: their mechanism of operation remains unknown. Here we explore the hyp...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Neural Networks
سال: 2020
ISSN: 0893-6080
DOI: 10.1016/j.neunet.2020.08.022